Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: multichain-testing #9462

Merged
merged 13 commits into from
Jun 13, 2024
Merged

feat: multichain-testing #9462

merged 13 commits into from
Jun 13, 2024

Conversation

0xpatrickdev
Copy link
Member

@0xpatrickdev 0xpatrickdev commented Jun 6, 2024

closes: #XXXX
refs: #8896

Description

  • Creates @agoric/multichain-testing package outside of the yarn workspace
  • Provides a containerized, multi-chain testing environment for local and CI testing using cosmology-tech/starship
    • The current config.yaml includes agoric, osmosis, cosmos, and hermes relayers between each. A chain registry (served over http), faucet, and block explorer are also provided).
  • Provides an ava test setup for accessing a chain registry, creating wallets, and requesting faucet funds
  • Ports utilities from @agoric/synthetic-chain and dapp-agoric-basics that help towards a smart wallet client that can execute offers.

Security Considerations

Scaling Considerations

Taking on some tech debt here wrt smart wallet utilities and being DRY, but we plan to address this in future. See #8963

Documentation Considerations

README.md documentation for running the service is provided.

Testing Considerations

The goal of this PR is to build greater confidence in our software via automated testing with fully-simulated chains.

Upgrade Considerations

Copy link

cloudflare-workers-and-pages bot commented Jun 6, 2024

Deploying agoric-sdk with  Cloudflare Pages  Cloudflare Pages

Latest commit: 54f453f
Status: ✅  Deploy successful!
Preview URL: https://9b06c957.agoric-sdk.pages.dev
Branch Preview URL: https://pc-8896-orchestration-docker.agoric-sdk.pages.dev

View logs

@0xpatrickdev 0xpatrickdev force-pushed the pc/8896-orchestration-docker branch 5 times, most recently from 0dd975c to ad76fce Compare June 6, 2024 20:20
@0xpatrickdev 0xpatrickdev force-pushed the pc/8896-orchestration-docker branch 2 times, most recently from baa17f1 to 4f482a6 Compare June 7, 2024 21:16
on:
# for now, only run on-demand
workflow_dispatch:
pull_request:
Copy link
Member Author

@0xpatrickdev 0xpatrickdev Jun 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO remove this line before merging

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you want it to run on every PR? consider running it the same times as the docker build tests (merges to master or with force:integration enabled)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated to the conditions for docker.yml.. unclear to me if force:integration will pick this up but agree with the suggested strategy

@0xpatrickdev 0xpatrickdev force-pushed the pc/8896-orchestration-docker branch 2 times, most recently from 75928b7 to 1dcc9d4 Compare June 7, 2024 22:10
Copy link
Member

@turadg turadg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of progress!

Since this is a testing framework, the primary thing is that it works and covers what we expect it to. Secondary is DX (e.g. minimizing surprises) which I think this PR should try to minimize. Third is maintainability, which I think we can punt to a follow-up after we have more experience with the tools

on:
# for now, only run on-demand
workflow_dispatch:
pull_request:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you want it to run on every PR? consider running it the same times as the docker build tests (merges to master or with force:integration enabled)

path: ./agoric-sdk

- name: Enable Corepack
run: corepack enable || sudo corepack enable
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pretty sure we should never need sudo in GH actions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had source from here but since removed

runs-on: ubuntu-latest-16core
strategy:
matrix:
node-version: ['18.x']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why mess with a matrix?

const urls = [];
config.chains?.forEach((chain) => {
- urls.push(`${registryUrl}/chains/${chain.name}`, `${registryUrl}/chains/${chain.name}/assets`);
+ urls.push(`${registryUrl}/chains/${chain.id}`, `${registryUrl}/chains/${chain.id}/assets`);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a bug in starship? if so, can we get a fix upstream?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there was a breaking change in the 0.2.x helm chart release. I opened an issue: cosmology-tech/StarshipJS#3

Comment on lines 12 to 20
const deleteKeys = async (deleteKey: (name: string) => Promise<string>) => {
for (const key of KEYS) {
try {
await deleteKey(key);
} catch (_e) {
// ignore
}
}
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You really want delete to fail silently?

Suggested change
const deleteKeys = async (deleteKey: (name: string) => Promise<string>) => {
for (const key of KEYS) {
try {
await deleteKey(key);
} catch (_e) {
// ignore
}
}
};
const deleteKeys = (deleteKey: (name: string) => Promise<string>) =>
Promise.all(KEYS.map(key => deleteKey(key).catch()));

Copy link
Member Author

@0xpatrickdev 0xpatrickdev Jun 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was helpful when I was working out the correct pattern here. I've since refactored in support.ts to:

const makeKeyring: (e2eTools: Pick<E2ETools, "addKey" | "deleteKey">) => Promise<{
    setupTestKeys: (keys?: string[]) => Promise<Record<string, string>>;
    deleteTestKeys: () => Promise<string[]>;
}>

If a test needs keys it can call setupTestKeys in test.before() and .deleteTestKeys() in test.after()


const bondDenom =
useChain('osmosis').chain.staking?.staking_tokens?.[0].denom;
if (!bondDenom) throw new Error('Bond denom not found.');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (!bondDenom) throw new Error('Bond denom not found.');
t.truthy(bondDenom, 'Bond demon not found');

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to this to appease to the TS compiler:

if (!bondDenom) {
    t.fail('Bond denom not found.');
  } else {
    const { balance } = await queryClient.queryBalance(addr, bondDenom);
    t.deepEqual(balance, { denom: bondDenom, amount: '10000000000' });
  }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here's hoping we get to Ava 6 soon which terminates on failed assertions, thus narrowing the code following

multichain-testing/test/walletFun.test.ts Outdated Show resolved Hide resolved

const { freeze } = Object;

const agdBinary = 'kubectl';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please rename to be accurate. e.g. kubectlBinary

/**
* @param {{ execFileSync: ExecSync }} io
*/
export const makeAgd = ({ execFileSync }) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll want to DRY this with other helpers and I'd also like to simplify it. but this works for now

export const pathToKey = (path) => path.join('.');

/** @param {string} key */
export const keyToPath = (key) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little surprised we don't have exports that do this already

@0xpatrickdev 0xpatrickdev force-pushed the pc/8896-orchestration-docker branch 2 times, most recently from 09ea9b9 to 8df0cb5 Compare June 11, 2024 02:00
@0xpatrickdev 0xpatrickdev marked this pull request as ready for review June 11, 2024 13:24
@0xpatrickdev 0xpatrickdev requested a review from turadg June 11, 2024 15:11
Copy link
Contributor

@raphdev raphdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I followed the docs and got it working with a bit of debugging with @0xpatrickdev. Some minor suggestions for the missing bits.

```bash
make fund-provision-poool
```

Copy link
Contributor

@raphdev raphdev Jun 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following this guide everything works except the two steps below. Would be helpful to provide an example:

Suggested change
For the steps below, you must import a key to `agd` or create a new one. For example, we can generate one to use as `ADDR` in the next steps:
`kubectl exec -i agoriclocal-genesis-0 -c validator -- agd keys add user1`

I considered adding a make target for adding a new key instead of the kubectl command, but not sure if it's useful beyond one-offs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, added this note as a note

Comment on lines 48 to 49
ADDR=agoric123 COIN=100000ubld make fund-wallet
ADDR=agoric123 make provision-smart-wallet
Copy link
Contributor

@raphdev raphdev Jun 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think because these are make vars, not envvars, they did not work as is in my shell. I think the correct COIN amount is 10000000ubld. I needed to instead do:

Suggested change
ADDR=agoric123 COIN=100000ubld make fund-wallet
ADDR=agoric123 make provision-smart-wallet
make fund-wallet ADDR=agoric123 COIN=10000000ubld
make provision-smart-wallet ADDR=agoric123

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thank you. I've applied these changes. Also upped to 20000000ubld so there's 10 BLD left over for gas.

Comment on lines 79 to 81

provision-smart-wallet:
kubectl exec -i agoriclocal-genesis-0 -c validator -- agd tx swingset provision-one wallet $(ADDR) SMART_WALLET --from faucet -y -b block
Copy link
Contributor

@raphdev raphdev Jun 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This did not work for me because --from key needs to match ADDR. We need to pass whatever key name matches ADDR:

Suggested change
provision-smart-wallet:
kubectl exec -i agoriclocal-genesis-0 -c validator -- agd tx swingset provision-one wallet $(ADDR) SMART_WALLET --from faucet -y -b block
KEY_NAME=user1
provision-smart-wallet:
kubectl exec -i agoriclocal-genesis-0 -c validator -- agd tx swingset provision-one wallet $(ADDR) SMART_WALLET --from $(KEY_NAME) -y -b block

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks - fixed

Copy link
Member

@turadg turadg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. I don't love the copy-pasta but since it does have an issue I agree it's appropriate to punt.

I'm requesting the package.json changes because that's a risk to merge.

Other than that, I judge this mostly by whether it improves the testing situation. It clearly does and I expect we'll continue to iterate.

multichain-testing/LICENSE Outdated Show resolved Hide resolved
Comment on lines 2 to 3
"name": "@agoric/multichain-testing",
"version": "0.1.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't be publishing this, so it doesn't need a version. We shouldn't be importing from it so it doesn't need a name.

It does need this to prevent publishing:

Suggested change
"name": "@agoric/multichain-testing",
"version": "0.1.0",
"private": true,

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was thinking external developers might want to depend on this at some point, but suppose it could be viewed as boilerplate instead. Can cross that bridge when we come to it.

multichain-testing/package.json Outdated Show resolved Hide resolved
"prettier": "^3.2.4",
"starshipjs": "2.0.0",
"tsimp": "^2.0.10",
"typescript": "^5.3.3"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please match the rest of the repo

Suggested change
"typescript": "^5.3.3"
"typescript": "^5.5.0-beta"

Not a big deal for the other deps but it'll be surprising to a developer if some code works in one place and not the other

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree it would be nice. 5.5.0-beta doesn't seem to play nicely here for whatever reason

"@types/node": "^20.11.13",
"@typescript-eslint/eslint-plugin": "^6.20.0",
"@typescript-eslint/parser": "^6.20.0",
"ava": "^5.3.0",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider using Ava 6 at this outset, so there's not more work to migrate later

const osmosisFaucet = useChain('osmosis').creditFromFaucet;
t.log('Requeting faucet funds');
await osmosisFaucet(addr);
await sleep(1000, t.log); // needed to avoid race condition
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please document with an XXX to help someone debugging the race or who wants to clean it up

Comment on lines 38 to 39
if (!bondDenom) {
t.fail('Bond denom not found.');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (!bondDenom) {
t.fail('Bond denom not found.');
t.truthy(bondDenom, 'Bond denom not found.');

would be more clear. I imagine you used the if to get type narrowing for the queryBalance call. This is a reason to adopt Ava 6. IIUC its assertions also do type assertions so lines after t.truthywill know bondDenom is not nullish.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ava 6. IIUC its assertions also do type assertions so lines after t.truthywill know bondDenom is not nullish.

No luck here, seemingly. I used the non-null-assertion-operator to make this more readable and drop the if/else

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

confirmed, though I can't make sense of it. There clearly is a type assertion.
Screenshot 2024-06-12 at 1 07 05 PM

@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2024 Interweb, Inc.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well done to include


const { freeze } = Object;

const agdBinary = 'kubectl';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please rename to be accurate. e.g. kubectlBinary

multichain-testing/tools/sleep.ts Outdated Show resolved Hide resolved
multichain-testing/tools/sleep.ts Outdated Show resolved Hide resolved
Comment on lines 57 to 60
"serial": true,
"environmentVariables": {
"LOCKDOWN_STACK_FILTERING": "verbose"
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wondered if we should do this throughout the repo, but I think this accomplishes the same and more:

Suggested change
"serial": true,
"environmentVariables": {
"LOCKDOWN_STACK_FILTERING": "verbose"
}
"require": [
"@endo/init/debug.js"
],
"serial": true

@0xpatrickdev 0xpatrickdev added the automerge:rebase Automatically rebase updates, then merge label Jun 12, 2024
@mergify mergify bot merged commit 3e597a8 into master Jun 13, 2024
63 checks passed
@mergify mergify bot deleted the pc/8896-orchestration-docker branch June 13, 2024 01:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge:rebase Automatically rebase updates, then merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants